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Abstract 

Background: A continuous growth in the publication of research papers means that there is an expanding volume 
of data available to the systematic reviewer. Sometimes, researchers can become overwhelmed by the sheer volume 
of data being processed, leading to inefficient data extraction. This paper seeks to address this problem by proposing a 
modification to the current systematic review methodology. 

Proposed method: This paper details the routine piloting of a systematic review all the way through to 
evidence-synthesis stage using data from a sample of included papers. 

Results and discussion: The result of piloting a sample of papers through to evidence-synthesis stage is to produce 
a 'mini systematic review'. Insights from such a pilot review may be used to modify the criteria in the data extraction 
form. It is proposed that this approach will ensure that in the full review the most useful and relevant information is 
extracted from all the papers in one phase without needing to re-visit the individual papers at a later stage. 

Conclusions: Routine piloting in systematic reviews has been developed in response to advances in information 
technology and the subsequent increase in rapid access to clinical papers and data. It is proposed that the routine 
piloting of large systematic reviews will enable themes and meaning in the data to become apparent early in the review 
process. This, in turn, will facilitate the efficient extraction of data from all the papers in the full review. It is proposed that 
this approach will result in increased validity of the review, with potential benefits for increasing efficiency. 
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Background 

The aim of research synthesis, or systematic reviewing, 
is to arrive at a more comprehensive and trustworthy 
picture of the topic being studied than is possible from 
individual pieces of research [1]. Systematic reviewing is 
a relatively young and rapidly developing field of study, 
with current best practice defined by guidelines such as 
those from the Centre for Reviews and Dissemination in 
the University of York and the Cochrane Handbook [2,3]. 
However, people embarking on systematic reviews still 
have many challenges to overcome, particularly relating 
to resource constraints. Reviews are major outputs of 
research and require time and resources, and tensions often 
arise when conducting a methodologically rigorous system- 
atic review with a tightly defined timeline or budget [4]. 
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A continuous growth in the publication of research 
papers, together with advances in the ability to identify 
published reports of controlled trials and information 
about on-going and unpublished studies in registers, 
means that there is a constantly expanding volume of 
research papers available [5-7]. Exhaustive searching can 
reveal high numbers of potentially relevant papers, and 
each paper must be processed and, if eligible, data extracted 
and assessed. Researchers might need to extract large 
volumes of data from the original articles during the data 
extraction stage. Sometimes, researchers can become over- 
whelmed by the sheer volume of data being processed. 
Often, it is not until the final synthesis stage (when data 
are fully tabulated or examined as a forest plot) that the 
patterns and relationships across the extracted data 
emerge, and the implications of the findings from each 
study, and from the review as a whole, become fully 
understood. At this late stage, researchers might realise 
that they have not extracted all the information they need 
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in order to fully answer their research question and will 
need to repeat earlier steps. On the other hand, they 
may realise that the data they have spent considerable 
time extracting is not in fact required. Either way, time 
(and money) may be lost by inefficient extraction of the 
relevant data. 

This paper seeks to address this problem by proposing 
a modification to the current systematic review method- 
ology, to take account of the rising volumes of clinical 
trial data available now and increasingly in the future. 

Main text 

At present, it is standard practice to pilot various stages in 
the systematic review process. Piloting of search strategies 
(scoping searches'), piloting of inclusion/exclusion criteria 
(within the screening stage) and piloting of data extraction 
tables or quality assessment checklists are all regularly 
employed by systematic reviewers [3]. In order to maxi- 
mise efficiency and minimise error when conducting large 
systematic reviews, this paper proposes extending such 
piloting to include all stages of the review process. In 
essence, this paper proposes conducting a mini systematic 
review! all the way through to evidence synthesis, on a 
sample of included studies. Insights from such a synthesis 
may be used to inform and refine data extraction in the 
full systematic review. It is proposed that such a routine 
modification of the systematic review process will result 
in an overall decrease in error and time to completion 
of the full review, even when time for piloting is taken 
into consideration. Another benefit for undertaking piloting 
across the entire review process may be that unforeseen 
problems with a review may be highlighted early on in the 
review process. This could then lead to a timely re-visiting 
and modification of the scope, the question or the ambition 
of the review, and may even support well-informed deci- 
sions to stop working on some reviews. 

This paper proposes to routinely pilot the systematic 
review process all the way through to evidence synthesis 
using data from a sample of included papers, but stopping 
short of the interpretation of the data to prepare implica- 
tions for future practice and research. Extracted data from 
the sample papers will be processed up to the synthesis 
stage before embarking on the full systematic review. 
Insights from the pilot synthesis stage may be used to 
modify criteria in the data extraction form, ensuring that 
only information needed to answer the review question is 
extracted from all the included studies. Such a modification 
will ensure that the most useful and relevant information is 
extracted from all the papers in a single phase without 
needing to re-visit the individual papers at a later stage. 
The proposed modification to the standard methodology 
hence ensures that resources are more effectively managed, 
in order to help ensure that the review question is answered 
within time and budget. 



Proposed method 

Traditionally, a detailed review protocol which clearly 
states the question to be addressed, the subgroups of 
interest and the methods and criteria to be employed for 
identifying and selecting relevant studies and extracting 
and analysing information, is prepared in advance [3,8]. 
This is important to avoid bias being introduced by deci- 
sions that are influenced by the accumulating data. Piloting 
of data extraction tables and quality appraisal forms is 
also routinely used to ensure that the information to be 
extracted is both standardised and relevant. It is common 
practice to take a small sample (e.g. 10) of included papers 
and use these to pilot data extraction and quality appraisal. 
This paper proposes an extension to current systematic 
review methodology, in which a full 'trial run' (or pilot) of 
the review would be done on such a sample of papers, 
resulting in a mini systematic review'. 

It is proposed that for a review with 20 or more 
included studies, the method would require a purposive 
sample of up to 10 papers. These 10 papers would be 
selected from the pool of included trials by the review 
authors and would be specifically chosen to cover a wide 
range of parameters for the review, such as outcome 
measure scales and/or time points, found in the literature 
of interest. By taking this sample of papers through each 
stage of the review process, we can estimate, at an early 
stage, the potential of the extracted data to answer the 
review question. Data extraction forms may then be refined 
and amended following the sample scoping exercise, ensur- 
ing efficient data extraction of all the studies to be included 
in the review. 

Specifically, it is proposed that current guidelines in 
conducting systematic reviews are followed up to the 
stage where papers have been assessed for inclusion in 
the review [2,3]. The pilot method is then employed, and 
a purposive sample (reporting a wide range of outcome 
measure scales and time points) of included papers is 
chosen from the total number of included studies to 
pilot up to the synthesis stage (step 1). Data from the 
sample papers are extracted (step 2) and critically appraised 
for quality and validity (step 3), and a sample synthesis is 
then performed (step 4). Results from this pilot synthesis 
are then used to inform modification of data extraction 
forms in the light of these preliminary synthesis findings 
(step 5), to ensure efficient and meaningful extraction of 
data from all included papers. By documenting the stages 
clearly in a study flow diagram (see Figure 1), the proposed 
methodological amendment is systematic, explicit and 
reproducible, thus minimising data dredging (where a 
mass of trial data is tested post hoc in order to find signifi- 
cant results and associations) and reducing bias. Strengths 
and weaknesses of such an approach are given in Table 1, 
and a theoretical case study of how the piloting process 
may be usefully employed is given below: 
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Included 

n=41 papers (36 individual studies) 



Pilot paper sample (e.g. n=10) 



Sample Data Extraction / Checking 

Study info and outcome data (extracted 
from sample of publications or obtained 
from primary researchers) checked and 
verified 



Appraise/ Analyse Sample Studies 

Sample study reports are critically ap- 
praised to assess quality and validity. 
Individual study data analysed 



Sample Synthesis 

Similarities and differences among study 
findings identified, described and investi- 
gated. Formal data pooling (meta- 
analysis) if appropriate. 




Refine Data Extraction /Checking 

Extracted study info / outcomes of in- 
terest amended if necessary in light of 
pilot synthesis findings. 



Full Data Extraction / Checking 

Study info and outcome data (finalised) 
checked and verified 



Appraise / Analyse All Individual Studies 

All study reports (n=36) are critically ap- 
praised to assess quality and validity. 
Individual study data analysed 



Full Synthesis 

Similarities and differences among all 
study findings identified, described and 
investigated. Formal data pooling (meta- 
analysis) if appropriate. 



Dissemination 



Figure 1 Theoretical process model for the pilot method. 
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Table 1 Strengths and weaknesses of piloting systematic reviews prior to full review 



Advantages 



Weaknesses 



Efficient and potentially time-saving when 
processing large numbers of studies 

Greater flexibility for optimally efficient 
data extraction 



Can be used in large umbrella overviews 
(a 'review of systematic reviews') 



Not necessary for reviews with small numbers of included studies. 

Need to have access to most of the potentially eligible studies in order to draw the 
purposive sample. If the gathering of studies for a review is taking place over a long 
time period, e.g. months, it might not be possible to identify the sample to start with. 

Not necessary for overviews with small numbers of included systematic reviews. 



Theoretical case study 

A systematic review of adults with type 2 diabetes explor- 
ing the association between biomedical outcomes (e.g. 
HbAlc, BMI/weight and blood pressure) and quality of 
life (including low mood and depression) following a 
diabetic drug intervention. Extraction of quantitative data 
would be performed using a structured data extraction 
form to include key study details, patient characteristics, 
diabetes-related factors, intervention, setting and outcome 
measures. It may emerge through the piloting process that 
in addition to baseline diagnosis of depression, a patient's 
history of depression is important in predicting changes in 
blood glucose levels (as measured by HbAlc), and so, the 
data extraction form would need to be modified to extract 
history of depression data from all the review papers. It 
may also emerge during the piloting process that some 
studies record single follow-up points for biomedical out- 
come measures, while others have multiple follow-up time 
points. After consideration of the pilot synthesis stage, the 
data extraction form could be modified to ensure that only 
the most clinically relevant time points required to answer 
the review question are extracted in the full review. 

Discussion 

This paper describes a new process for routine piloting 
within large systematic reviews. It is proposed that the 
method offers a timely evolution of established systematic 
review methodology. It has been developed in response to 
advances in information technology and the subsequent 
increase in rapid access to clinical papers and data. The 
need to efficiently process increasing volumes of clinical 
trial data is set to continue into the future, and it is pro- 
posed that the routine piloting of large systematic reviews 
will enable patterns and relationships in the data to 
become apparent early in the review process. This, in turn, 
will facilitate the efficient extraction of data from all the 
papers in the full review. It is proposed that this approach 
will result in increased external validity of the review, with 
potential benefits for increasing efficiency, particularly for 
systematic review teams who have methodological expert- 
ise, but no clinical experience of the health intervention 
under review. Systematic reviewers involved in health 
technology assessment (HTA) work, where researchers 
regularly review different clinical areas, may particularly 



benefit from this approach. In addition, the method may 
be beneficial if undertaking a systematic review where 
consideration of generalizability of randomised trials is of 
particular interest [9]. This may be particularly important 
where the review teams have concerns about the potential 
lack of generalizability of included trials. In such reviews, 
efficient extraction of data on the intervention process 
and context may benefit most from a rapid assessment 
early in the review process, in order for a comprehensive 
and meaningful analysis to be undertaken. Cochrane sys- 
tematic review teams tend to have strong expertise in the 
clinical area they are reviewing and are hence more able 
to define a priori the outcomes and scales of most interest 
to both the patient and the clinician. It is hence likely that 
there may not be such a need to use the routine piloting 
method to increase review validity. However, the possibil- 
ity remains that Cochrane teams engaged in a large-scale 
review or review of reviews' may benefit from the method 
in order to increase efficiency of data extraction. Evalu- 
ation of the method in different contexts is hence needed. 

Routine piloting of systematic reviews is one response 
to the need for increased efficiency in an age of increasing 
access to clinical trial data. Other approaches to increase 
efficiency involve semantic technologies to automate part 
of the review process and include text mining (The 
Evidence for Policy and Practice Information and Co- 
ordinating Centre (EPPI-Centre)) [10], Systematic Review 
Data Repositories [11,12] and automation of the produc- 
tion of (Cochrane) protocols [13,14]. More recently, the 
complete automation of systematic reviews has been 
proposed [4,13,15]. While offering tremendous potential 
benefits for increased efficiency, there are some concerns 
relating to interpretation of language and extrapolation of 
meaning in fully automated reviews [16,17]. Interpreting 
data is a crucial aspect in a review, and critics of automa- 
tion in systematic reviews state that human judgement 
and interpretation are irreplaceable in order to derive 
meaning from the evidence synthesised and draw clinically 
relevant conclusions, especially from pooled data [16]. 
In addition, human involvement is necessary in order 
to contact authors to assess if a study meets the inclusion 
criteria of a review and for the evaluation of risk of bias 
when reporting of original studies is not complete or 
precise [16-18]. Others point out that given the often 
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inconsistent and incomplete descriptions of public health 
interventions, interpretation plays a crucial role when 
re-describing, classifying or quantifying health interven- 
tions, and this might best be done by humans [19,20]. 

Conclusions 

Routine full piloting of systematic reviews may offer a 
timely addition to the systematic reviewer's methodology 
toolkit, whereby human engagement in the review process 
ensures that meaningful interpretation of results and hence 
validity (clinical relevance) of the review are maximised, 
while efficiency is potentially increased. In addition to 
potentially maximising review validity and efficiency, the 
method might be usefully employed in emerging areas of 
systematic review methodology, e.g. reviews of reviews' 
and generalizability of randomised controlled trials, where 
clear and detailed methodological guidance is yet to be 
fully established. 

It is hoped that this paper will serve as stimulation for 
further discourse on the subject of maximising validity 
and efficiency in systematic reviews, given the increasing 
volume of research papers available both now and in the 
future. Details of approaches developed by other research 
teams to address this issue, or evaluation of the above 
routine piloting method, would be most welcome. 
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